Reinforcement Learning Policy Approximation by Behavior Trees

Traditionally a Reinforcement Learning (RL) policy is stored in a lookup table. From such a table it is difficult to observe the behavioral logic or manually adjust this logic post-learning is difficult. This paper shows how behavioral logic of a RL controller is presented in an insightful manner and can be adjusted using the Behavior Tree (BT) framework. It shows a method to approximate an RL policy using a Genetic Algorithms (GA) for BTs for a guidance task carried out by an UAV navigating in an unknown environment. The method shows how Discrete Time Markov Chains (DTMCs) can be used to increase optimization speed. The execution of the RL controller that interacts with the environment is mapped to a DTMC, which results in a representation of the one-step transition matrix. The GA evaluates BTs by transitioning through this DTMC instead of the simulation environment. Without the need to simulate a computationally expensive environment, the optimization is performed faster. The method is demonstrated on a UAV simulation using a Q-learning algorithm to learn a guidance task. The guidance task consists of an avoidance behavior and goal-seek behavior. The evolved BT, containing 6 nodes, successfully identifies 66% of the correct actions of the RL policy for all states in the Q-table. When this BT is run in the simulation environment it results in a success rate of 93%. After adaptation by the experimenter the success rate for all states in the Q-table increases to 86% and in simulation to 96%. For the verification of the avoidance behavior with the evolved BT only 3 nodes need to be verified, compared to 1448 state-action pairs of the Q-table.